Parallel MPI implementation of training algorithms for medium-size feedforward neural networks
نویسندگان
چکیده
Artificial neural networks provide a feasible approach to model complex engineering systems. Computational parallelism is assumed as a basis of the neural architectures. In the Russian Federal Nuclear Center VNIITF there exists a neural simulator Nimfa. In the framework of this project parallel versions of training algorithms for feed-forward neural networks based on the MPI standard are developed. In this paper we present our experience with this implementations on shared-memory multiprocessors and on Linux PC clusters. Parallelism of Neural Computations Parallelism is one of the underlying principles of the artificial neural networks [1-7]. Several parallel schemes are proposed in literature. So far, some of them are implemented in neural hardware – in the comparatively inexpensive neural boards. Another approach is to exploit the inherent parallelism and to map them on specialpurpose hardware or to simulate neural networks on general-purpose parallel computers. We focus on the data parallel simulation of ANN. ANN consists of a number of very simple units, which are connected with each other via adjustable links. In the feed-forward networks, units are grouped into layers with continuous activation functions. We present a set of input-target (desired output) examples to the network representing the functional relationship to be learned. Supervised learning is an optimization problem. The task is to minimize a cost function or error measure, i.e. the “difference” between the actual output and the desired output. During training, the parameters of the ANN are adapted in an iterative process.
منابع مشابه
Weight Perturbation: An Optimal Architecture and Learning Technique for Analog VLSI Feedforward and Recurrent Multilayer Networks
Previous work on analog VLSI implementation of multilayer perceptrons with on-chip learning has mainly targeted the implementation of algorithms such as back-propagation. Although back-propagation is efficient, its implementation in analog VLSI requires excessive computational hardware. It is shown that using gradient descent with direct approximation of the gradient instead of back-propagation...
متن کاملParallel Gradient Descent for Multilayer Feedforward Neural Networks
We present a parallel approach to classification using neural networks as the hypothesis class. Neural networks can have millions of parameters and learning the optimum value of all parameters from huge datasets in a serial implementation can be a very time consuming task. In this work, we have implemented parallel gradient descent to train multilayer feedforward neural networks. Specifically, ...
متن کاملA Reconfigurable Computer REOMP
This work describes a proposal of reconfigurable computer, and their appl ication to hardware implementations of neural networks. Although lhe neural network funclions correspond lo the brain functions, our computer is based on the current technology, which is completely di fferent from the internai structure of the brain based on the neuronal cells. The proposed Reconfigurable Orthogonal Multi...
متن کاملConstructive Training Methods for feedforward Neural Networks with Binary weights
Quantization of the parameters of a Perceptron is a central problem in hardware implementation of neural networks using a numerical technology. A neural model with each weight limited to a small integer range will require little surface of silicon. Moreover, according to Occam's razor principle, better generalization abilities can be expected from a simpler computational model. The price to pay...
متن کاملParallel Levenberg-Marquardt-Based Neural Network Training on Linux Clusters - A Case Study
This paper addresses the problem of pattern classification using neural networks. Applying neural network classifiers for classifying a large volume of high dimensional data is a difficult task as the training process is computationally expensive. A parallel implementation of the known training paradigms offers a feasible solution to the problem. By exploiting the massively parallel structure o...
متن کامل